One of the problems facing architects and providers of communication networks and related services is the provision of network security which has the ability to effectively detect and isolate malicious traffic in the network before its effect is felt by the intended recipients. Malicious traffic can take the form of attacks against a service provider's network equipment, or attacks directed at external nodes, but which pass through a service provider's network. Service providers need to detect and eliminate this traffic in order to protect both their routers and the nodes connected to their network, which often belong to their customers. The difficulty in detecting these attacks arises from the fact that there are many different varieties of malicious traffic, and there are limited resources available at the router for traffic monitoring. A further difficulty is that the type of attacks are continually changing, and in general cannot be isolated by simple filters.
Currently, there are two primary means to provide network security focused monitoring required for general attack detection: (1) the use of special purpose in-line appliances, an example of which is shown in FIG. 1, and (2) the use of existing router monitoring abilities to generate statistics that are interpreted by off-node tools, as for example shown in FIG. 2.
Referring to FIG. 1, a communication network 1 comprises a core routed network 3 and edge routers 5 of other communication networks, each of which is connected to the core routed network 3 through an in-line security appliance 7. The in-line security appliances monitor traffic flowing between the edge routers 5 and core network 3 for the presence of malicious traffic. If malicious traffic is detected, the in-line security appliances attempt to isolate the traffic to prevent it migrating from one network to another.
Special purpose network appliances for attack detection are common in enterprise Local Area Networks (LANs). Examples of in-line intrusion detection systems include products from Checkpoint Software Technologies and Juniper Networks NetScreen Appliance Line. These devices, however, are very rarely deployed in large scale networks, i.e. service provider Wide Area Networks (WANs). The number and speed of interfaces in wide area networks lead to a relatively high cost per bit, and provide a challenge that in-line appliances cannot solve in an economical manner.
In the example of FIG. 2, the data flow through a router 9 is sampled and the data collected in a flow table 11. Designated packet information is selected from the flow table and exported to an external server 13, called a “collector”, for interpretation. The interpreted results are typically transferred to one or more application servers 15 for further use, such as monitoring by an operator. A collector and accompanying operational support system (OSS) software suite that analyzes, for malicious traffic, data collected at a network node is offered by Arbor Networks. Other off-line software tools exist that also analyze traffic flow records, examples of which include “Ntop”, which provides visualization of user selected aggregate groups, and “AutoFocus”, which automatically extracts “significant events” from the observed statistics.
In systems which use exported flow data sampled at network nodes, the stream of input data from the sampler of the node to the flow table is determined by the configuration of the flow statistics collection system at the router, which is provisioned manually by a network operator. The packet information (i.e. header fields) that is used to create the flow statistics for export is also determined when the data collection system is provisioned. The granularity of the collected statistics, both in terms of what flow information is extracted, and at what sampling rate packets are examined, is constant throughout the monitoring process.
A drawback of this approach is that in order to detect a malicious threat, large quantities of data must be collected and exported to the external collector for analysis. This requires both the use of a large amount of internal memory and processing cycles at the router to monitor, collect and export the required amount of data to enable detection of malicious traffic. This technique also consumes network resources, and in particular, available bandwidth in requiring routers to export large amounts of flow data to external servers. Furthermore, this solution usually requires many collectors to be deployed throughout the network in order to process the large volume of flow data produced by the routers.